The present invention is directed to a method and apparatus for maintaining consistency of data stored in a group of mirroring devices.
A number of computer system configurations mirror information stored in source storage devices to target storage devices that can be disposed at locations remote from the source storage devices. An example of such a computer system is a data mirroring system 100 discussed below in connection with FIG. 1.
In the data mirroring system 100 of FIG. 1, one or more host CPU""s 102 are coupled (via one or more communication links 114) to a source storage system 104. In the example shown, the source storage system 104 includes a plurality of xe2x80x9csourcexe2x80x9d storage devices 110a-f. The source storage devices 110a-f can be included in a single source storage system 104 as shown in FIG. 1, or multiple source storage systems 104 can be employed, and the source storage devices 110a-f can be distributed across the multiple source storage systems. Each source storage system 104 may, for example, comprise a storage system from the SYMMETRIX line of disc arrays available from EMC Corporation, Hopkinton, Mass.
The data stored in the source storage devices 110a-f may be crucial to the operation of the host CPU(s) 102. Therefore, a contingency solution is desirable in the event that a problem is experienced with any of the source storage devices 110a-f to ensure that the data stored thereby is not lost, and to minimize the risk of the host CPU(s) 102 being down due to problems with the source storage devices 110a-f. Potential problems with the source storage devices 110a-f may include, for example, hardware or software errors that may make stored data unrecoverable. Additionally, catastrophic events such as an earthquake or other natural disaster could result in the destruction of one or more of the source storage devices 110a-f. 
One solution for protecting the data stored in the source storage devices 110a-f is to mirror the data in a corresponding set of xe2x80x9ctargetxe2x80x9d storage devices. The data mirroring system 100 of FIG. 1 is an example of such a system, wherein the source storage system 104 is coupled (via links 112a-f) to a target storage system 106. The target storage system 106 includes a plurality of target storage devices 116a-f corresponding, respectively, to the source storage devices 110a-f in the source storage system 104. As data is written to any of the source storage devices 110a-f, it is also written to a corresponding one of the target storage devices 116a-f in the target storage system 106. If one of the source storage devices 110a-f is destroyed or experiences an error that renders stored data unrecoverable, the data can be retrieved from the corresponding one of the target storage devices 116a-f. The target storage devices 116a-f can be included in a single target storage system 106 as shown in FIG. 1, or multiple target storage systems 106 can be employed, and the target storage devices 116a-f can be distributed across the multiple target storage systems 106. As with the source storage system(s) 104, each of the target storage systems 106 may, for example, be a storage system from the SYMMETRIX line of disc arrays available from EMC Corporation, Hopkinton, Mass.
When each of the source and target storage systems 104 and 106 is implemented using one or more of the SYMMETRIX line of disk arrays available from EMC Corporation, a feature called SYMMETRIX Remote Data Facility (SRDF) can be employed to implement the connection therebetween. SRDF is described in numerous publications available from EMC Corporation, including the SYMMETRIX Remote Data Facility Product Manual, P/N 200-999-554, rev. B, June 1995. SRDF is also described in U.S. Pat. No. 5,544,347 (Yanai).
In the illustrative data mirroring system 100 of FIG. 1, first, second, and third sets of source storage devices 110a-b, 110c-d, and 110e-f, respectively, in the source storage system 104 are in communication (via links 112a-b, 112c-d, and 112e-f, respectively) with corresponding first, second, and third sets of target storage devices 116a-b, 116c-d, and 116e-f in the target storage system 106. The links 112 are referred to herein as xe2x80x9cmirroring links,xe2x80x9d and the term xe2x80x9cmirroring communicationxe2x80x9d is used herein to refer to communication between source and target storage systems 104 and 106 over the mirroring links 112 that permits the target storage devices 116a-f to mirror data stored by the source storage devices 110a-f. As shown in FIG. 1, the source storage devices 110a-b, the mirroring links 112a-b, and the target storage devices 116a-b constitute a first mirroring group 108a; the source storage devices 110c-d, the mirroring links 112c-d, and the target storage devices 116c-d constitute a second mirroring group 108b; and the source storage devices 110e-f, the mirroring links 112e-f, and the target storage devices 116e-f constitute a third mirroring group 108c. A xe2x80x9cmirroring groupxe2x80x9d is a group of storage devices in the source and target storage systems 104 and 106 between which data can be transferred via a common set of mirroring links 112. Generally, storage devices 110 and 116 do not share any mirroring links 112 with storage devices 110 and 116 in other mirroring groups 108.
To perform a write operation to the source storage system 104, a host CPU 102 transmits an xe2x80x9cI/O command chainxe2x80x9d to the source storage system 104. Each I/O command chain can include one or more commands directed to a particular logical volume stored by the source storage system 104. A logical volume is a unit of information that the host CPU 102 perceives to correspond to a single physical storage device, but that may be mapped within the source storage system 104 to physical storage space on one or more source storage devices 110. An I/O command chain for a write operation (xe2x80x9ca write-requesting command chainxe2x80x9d) includes various instructions for the source storage system 104 receiving the I/O command chain, as well as the data to be written during the operation. The host CPU 102 generally receives only a single message back from the source storage system 104 in response to its sending of a write-requesting command chain. This message generally indicates either: (1) that the I/O command chain has been xe2x80x9ccommittedxe2x80x9d (i.e., processed successfully) by the source storage system 104, or (2) that an error occurred preventing the I/O command chain from being committed by the source storage system 104. Typically, the source storage system 104 will not provide the xe2x80x9cI/O command chain committedxe2x80x9d message back to the host CPU 102 unless and until the data has been written successfully to one or more of the source storage devices 110, or to a temporary storage space (such as cache) in the source storage system 104.
One of the purposes for mirroring data in the target storage system 106 is that if the data on the source storage system 104 is rendered unrecoverable, the data can be retrieved from the target storage system 106. Thus, it is desirable to ensure that the data stored in the target storage system 106 is internally consistent, and reflects an accurate mirror of the data stored in the source storage system 104 at some particular point in time. If the data in the target storage system 106 does not represent an accurate mirror of the data in the source storage system 104 for a given point in time, the data in the target storage system 106 cannot be reloaded onto the source storage system 104 to place the source storage system 104 back into a valid state. For example, if the data stored in the source storage system 104 and mirrored on the target storage system 106 is included in a database, but the information stored in target storage system 106 does not reflect an accurate picture of the database at a given point in time, then if an error is encountered that prevents the database from being recovered from the source storage system 104, the target storage system 106 cannot be used to recover the database because the data stored thereby is not an accurate reflection of the state of the database at any particular point in time.
In a data mirroring system such as the data mirroring system 100 shown in FIG. 1, Applicant""s have recognized that malfunctions with the mirroring links 112a-f that result in the loss of mirroring communication between the source and target storage systems 104 and 106 for one of the mirroring groups 108a-c can cause a data consistency problem to arise under certain circumstances. Suppose, for example, that all of the mirroring links 112a-b for the mirroring group 108a were to malfunction so that mirroring communication over the mirroring links 112a-b was disabled, thereby preventing updates to the source storage devices 110a-110b from being transferred to the target storage devices 116a-b. Suppose further that mirroring operations were to continue normally over the mirroring links 112c-d and 112e-f for the mirroring groups 108b and 108c, respectively, so that updates to the source storage devices 110c-f would continue to be transferred to the target storage devices 116c-f. If, under these circumstances, the data stored in either of the target storage devices 116a-b was logically related to the data stored in any of target storage devices 116c-f (e.g., if they stored potions of the same database), then the data stored in the target storage system 106 would not represent a valid state of the data stored in the source storage system 104 at any particular point in time. Therefore, if the source storage system 104 were to fail in the above-described situation, the data on the target storage system 106 could not be used to place the source storage system 104 back into a valid state for any particular point in time.
The above-described problem may be better understood in light of the following illustrative situation in which the host CPU(s) 102 write xe2x80x9cdependentxe2x80x9d data units to respective source storage devices 110 when mirroring communication is disabled for only one of the source storage devices 110 to which the dependent data units are being written. A second unit of data is dependent on a first unit of data if the second unit of data is a function of the first unit of data so that, if the first unit of data is altered, the second unit of data must also be altered if it is to accurately reflect the current value of the first unit of data. For example, if first, second, and third units of data represent values xe2x80x9cA,xe2x80x9d xe2x80x9cB,xe2x80x9d and xe2x80x9cA +B,xe2x80x9d respectively, then the third unit of data is dependent on both the first and the second units of data, whereas the first and second units of data are not dependent on one another.
Using this simple example of dependent data units, in the situation described above wherein mirroring communication is disabled over the mirroring links 112a-b but is enabled over the mirroring links 112c-f, if the host CPU(s) 102 initially write the first unit of data (i.e., the value xe2x80x9cAxe2x80x9d) to the source storage device 110a in the mirroring group 108a (xe2x80x9cthe disabled mirroring groupxe2x80x9d), and subsequently write the third unit of data (i.e., the value xe2x80x9cA+Bxe2x80x9d) to the source storage device 110c in the mirroring group 108b (xe2x80x9can operational mirroring groupxe2x80x9d), then both the first and third units of data would be written to the source storage system 104, but only the third unit of data (and not the first unit of data) would be written to the target storage system 106. As a result, the target storage system 106 would store an updated value of xe2x80x9cA+B,xe2x80x9d without also storing an updated value of xe2x80x9cA.xe2x80x9d This inconsistency can pose problems if it becomes necessary to recover the data from the target storage system 106.
One technique that has been used in the past to avoid the above-described data inconsistency problem has been to prevent the source storage system 104 from committing any write-requesting command chain from a host CPU 102 to a source storage device 110a-f if that source storage device 110a-f is a member of a mirroring group 108 that is unable to engage in mirroring communication across its mirroring links 112. Using this technique, since each write-requesting command chain directed to a source storage device 110 in a disabled mirroring group 108 is not committed by the source storage system 104, the host CPU 102 transmitting the I/O command chain will repeatedly attempt (unsuccessfully) to perform the write operation.
It should be appreciated that application programs executing on the host CPU(s) 102 typically will not begin execution of an instruction that is dependent upon a unit of data updated by a previous instruction until the previous instruction completes, which requires that the I/O command chain that implements the previous instruction has been committed by the source storage system 104. Therefore, by preventing write-requesting command chains directed to source storage devices 110a-f for which mirroring communication has been disabled from committing, it is ensured that no later instructions dependent on the data to be written by those write-requesting command claims will be executed, thereby avoiding the above-discussed data inconsistency problem. In this respect, the data stored in the target storage system 106 will be a valid mirror of the data stored in the source storage system 104 at the point that the mirroring group became disabled.
It should be appreciated that although the above-described technique avoids a data inconsistency problem, it does so by preventing I/O command chains directed to source storage devices 110 within a disabled mirroring group from ever completing. As a result, the CPU 102 that initiates the I/O command chain will repeatedly attempt (unsuccessfully) to execute the command chain, eventually resulting in a crash of the application or other program that includes the command chain.
According to one aspect of the present invention, a method for use in a data mirroring system involves disabling mirroring communication from a second source storage device to a second target storage device when mirroring communication from a first source storage device to a first target storage device is disabled.
According to another aspect of the present invention, a source storage system for use in a data mirroring system includes at least first and second source storage devices to be coupled to at least one host to enable the at least one host to perform write operations to the first and second source storage devices, and further to be coupled to at least first and second target storage devices, respectively, to enable mirroring communication from the at least first and second source storage devices to the at least first and second target storage devices, and at least one controller to, responsive to mirroring communication from the first source storage device to the first target storage device becoming disabled, disable mirroring communication from the second source storage device to the second target storage device.
According to another aspect of the invention, at least one computer-readable medium has a plurality of instructions stored thereon which, when executed by at least one processor included in a data mirroring system, cause the at least one processor perform a method including a step of disabling mirroring communication from a second source storage device to a second target storage device when mirroring communication from a first source storage device to a first target storage device is disabled.
According to another aspect of the present invention, a method for use in a data mirroring system involves instructing at least one source storage system to disable mirroring communication from a second source storage device to a second target storage device in response to at least one host receiving an indication that mirroring communication from a first source storage device to a first target storage device is disabled.
According to another aspect of the invention, at least one computer-readable medium has a plurality of instructions stored thereon which, when executed by at least one host processor included in a data mirroring system, cause the at least one host processor to perform a method including a step of instructing the at least one source storage system to disable mirroring communication from a second source storage device to a second target storage device in response to the at least one host processor receiving an indication that mirroring communication from a first source storage device to a first target storage device is disabled.
According to another aspect, a host for use in a data mirroring system includes at least one controller to be coupled to at least first and second source storage devices to perform write operations to the at least first and second source storage devices. The at least one controller is configured to, responsive to receiving an indication that mirroring communication from the first source storage device to a first target storage device is disabled, instruct at least one source storage system in which the at least first and second source storage devices are included to disable mirroring communication from the second source storage device to a second target storage device.
According to another aspect of the invention, a method for use in a data mirroring system involves storing information in the data mirroring system identifying at least one subset of a plurality of mirrored source storage devices for which a consistent representation of data is desired to be maintained on at least one corresponding subset of a plurality of target storage devices.
According to another aspect of the invention, a host for use in a data mirroring system includes at least one memory, and at least one controller to store information in the at least one memory identifying at least one subset of a plurality of mirrored source storage devices for which a consistent representation of data is desired to be maintained on at least one corresponding subset of a plurality of target storage devices.
According to another aspect of the invention, a source storage system for use in a data mirroring system includes at least one memory, and at least one controller to store information in the at least one memory identifying at least one subset of a plurality of mirrored source storage devices for which a consistent representation of data is desired to be maintained on at least one corresponding subset of a plurality of target storage devices.
According to another aspect of the invention, a method for use in a data mirroring system involves placing at least one of a plurality of mirrored source storage devices in one of a first state, a second state, and a third state. In response to a source storage system receiving a write-requesting command chain directed to the at least one of the plurality of mirrored source storage devices when the at least one of the plurality of mirrored source storage devices is in the first state, the command chain is permitted to commit and data written by the command chain is transferred to a corresponding one of a plurality of target storage devices. In response to the source storage system receiving a write-requesting command chain directed to the at least one of the plurality of mirrored source storage devices when the at least one of the plurality of mirrored source storage devices is in the second state, the command chain is prevented from committing. In response to the source storage system receiving a write-requesting command chain directed to the at least one of the plurality of mirrored source storage devices when the at least one of the plurality of mirrored source storage devices is in the third state, the command chain is permitted to commit without transferring data written by the command chain to the corresponding one of the plurality of target storage devices.
According to yet another aspect of the invention, a source storage system for use in a data mirroring system includes a plurality of mirrored source storage devices to be coupled to at least one host to permit the at least one host to perform write operations to the plurality of mirrored source storage devices, and further coupled to a plurality of target storage devices to enable mirroring communication from each of the plurality of mirrored source storage devices to a corresponding one of the plurality of target storage devices, and at least one controller to place at least one of the plurality of mirrored source storage devices in one of a first state, a second state, and a third state, wherein, in response to the source storage system receiving a write-requesting command chain directed to the at least one of the plurality of mirrored source storage devices when the at least one of the plurality of mirrored source storage devices is in the first state, the at least one controller permits the command chain to commit and transfers data written by the command chain to the corresponding one of the plurality of target storage devices, wherein, in response to the source storage system receiving a write-requesting command chain directed to the at least one of the plurality of mirrored source storage devices when the at least one of the plurality of mirrored source storage devices is in the second state, the at least one controller prevents the command chain from committing, and wherein, in response to the source storage system receiving a write-requesting command chain directed to the at least one of the plurality of mirrored source storage devices when the at least one of the plurality of mirrored source storage devices is in the third state, the at least one controller permits the command chain to commit without transferring data written by the command chain to the corresponding one of the plurality of target storage devices.
According to another aspect of the invention, at least one computer-readable has a plurality of instructions stored thereon which, when executed by at least one processor included in a data mirroring system, cause the at least one processor to perform a method including steps of: (A) placing at least one of a plurality of mirrored source storage devices in one of a first state, a second state, and a third state; (B1) in response to a source storage system receiving a write-requesting command chain directed to the at least one of the plurality of mirrored source storage devices when the at least one of the plurality of mirrored source storage devices is in the first state, permitting the command chain to commit and transferring data written by the command chain to a corresponding one of a plurality of target storage devices; (B2) in response to the source storage system receiving a write-requesting command chain directed to the at least one of the plurality of mirrored source storage devices when the at least one of the plurality of mirrored source storage devices is in the second state, preventing the command chain from committing; and (B3) in response to the source storage system receiving a write-requesting command chain directed to the at least one of the plurality of mirrored source storage devices when the at least one of the plurality of mirrored source storage devices is in the third state, permitting the command chain to commit without transferring data written by the command chain to a corresponding one of the plurality of target storage devices.